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Assessing the Impact of Curricular Reform - 
Measures of Course Efficiency and Effectiveness 



Because the typical approach of examining semester course grades as a measure of curricular or 
instructional change is not satisfactory, this paper outlines an alternative assessment model focusing on 
the academic success of gateway mathematics course students in the next course in the curricular 
sequence. The model was developed by a state university system Louis Stokes Alliance for Minority 
Participation, which is part of a nation-wide initiative funded by the National Science Foundation to 
increase the number of under-represented minority students graduating in science, engineering, 
mathematics, and technology (SMET fields) as part of its mandate from Congress.^ Engaging faculty in 
discussions about the kind of information that would be useful to them as they seek to improve student 
learning and academic achievement is the foundation of the model. 
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Assessing the Impact of Curricular Reform — 
Measures of Course Efficiency and Effectiveness 



> Retaining Minority Students in Science and Engineering Fields 

During their freshman and sophomore years in college, students of color have the greatest attrition rates 
among students majoring in science, mathematics, engineering and technology, or the SMET fields 
(Gainen, 1995). This occurs although they report grade point averages of A- or better upon college entry. 
One reason proposed by researchers for the high attrition rates in SMET fields among minority college 
students may be the classroom/teaching climate that makes introductory course material difficult for some 
students to grasp. According to Tobias (1990), introductory college courses have remained competitive, 
exceedingly difficult, and intimidating primarily because science professors anticipate that the current 
generation of high school science students is better trained. Therefore, instead of nourishing average 
students' skills and science interests, college instructors still expect to interact with and teach only the 
most accomplished students, almost ignoring the needs of potentially good students (e.g., B students). 

Thus, for many first-time college freshmen interested in science and engineering careers, the introductory 
mathematics and science courses take on a gatekeeper role (Tobias, 1992; Van Valkenburg, 1990). These 
initial courses frequently block students from progressing into degree programs, thus eliminating students 
who are judged as lacking the analytical ability to become competent scientists and engineers in light of 
their inability to pass the gatekeeper courses. Thus, a shortage of both majority and minority students 
results, also ensuring both a shortage of minority scientists and potential faculty members in SMET areas. 

Since the late 1980’s,the National Science Foundation (NSF) has sponsored a national debate about the 
shortage of scientists, mathematicians, engineers, and technicians who are U.S. citizens. Extensive 
discussions about curricular issues, instructional practices, the amount of information an instructor is 
expected to cover, and what kinds of student outcomes should be expected in gatekeeper courses have 
been supported. In response to this NSF initiative. The University of Texas System Louis Stokes 
Alliance for Minority Participation (UT-System LSAMP) was created as part of the effort to increase the 
number of under-represented minority students who graduate with SMET degrees. This partnership of 
universities and community colleges has invested considerable effort in improving their science and 
mathematics gatekeeper courses to help increase the success and retention of freshmen and sophomore 
students. One component involved the development of a model to assess the efficiency and effectiveness 
of gatekeeper courses at The University of Texas at El Paso (UTEP). 

This paper provides an overview of this ICE^ model: first, a summary of relevant literature concerning 
educational reform and evaluation efforts; second, findings from the model as applied to selected 
mathematics courses at UTEP; finally, an outline of how gateway course evaluation can support faculty 
curriculum and instructional change efforts. 

r- The Literature on Evaluation of Curricular Reform Efforts 

Science curricular reform literature contains data from a variety of evaluation methods. For example, 
some assessment efforts reported anecdotal evidence (Coppola, 1995; Magner, 1996). Others proposed 
effective evaluation methods without providing evaluation data (Dally & Zhang, 1993; Prabhu & 
Ramarapu, 1994; Seltzer, Hilbert, Robinson, & Swartz, 1996; Willemsen, 1995). In addition, some 
researchers used student satisfaction ratings of new courses as compared with reformed courses (Johnson 
& Leonard, 1994; Woods, 1996). A major NSF project to update engineering education published a 
curriculum innovation manual that emphasized the importance of quality improvement, but its contents 
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focus almost entirely on student feedback to improve how the course is delivered and its content 
(SUCCEED Project, 1996). 



Still others reported student course retention and failure rates (Felder et al„ 1993; Luck & Stephens, 1992; 
Osborne & Fullilove, 1993; Ratay, 1994). Some evaluation efforts used student course and final 
examination grades comparing original courses with reformed courses (Davis & McCoullum, 1992; 
Hershberger & Plantholt, 1994; Johnson, 1995; Lomen, 1992; Penn, 1994; Tidmore, 1994; Woods,’l996). 
Some studies attempt to incorporate aspects of several of these approaches (Felder, Felder, Mauney, 
Hamrin, & Dietz, 1995), including analysis of the match between students’ expectations and their actual 
experiences (Johnson & Leonard, 1994). Adelman (1989) published a manual of creative approaches to 
assessment of undergraduate learning in Biology, Chemistry, Computer Science, Mechanical 
Engineering, and Physics. Ratcliff (1994) has developed a coursework cluster analysis model for linking 
assessments of the general learning of graduating seniors with their previous coursework that can be 
adapted to examine gateway courses. Evaluation of the Emerging Scholars Program is an example of a 
systematic approach to examining the performance of minority students in mathematics (Moreno et al , 
1999) 

The evidence from evaluations of science and mathematics curricular reform efforts may be limited in 
two ways. First, some observers are quick to challenge approaches that compare course grades, 
questioning whether innovative professors may have eased the academic rigor of the course and/or 
lowered the standards, thereby fostering grade inflation (Rosen & Klein, 1996). For example, according 
to Wilson (1997), critics of the Harvard Calculus Model contend that those calculus courses have been 
“watered down” in an attempt to make Calculus more relevant to undergraduates. Secondly, student 
satisfaction ratings of the new courses do not provide an objective measure of students' academic 
attainment or future success. In general, many of the evaluation case studies are complex to manage and 
somewhat removed from faculty concerns about curriculum innovation and monitoring of student 
learning in gateway courses. Few offer a systemic approach to or model for the evaluation of curricular 
change. 

> Possible Student Outcomes in a Gateway Course 

Traditionally, faculty have looked at course grades (i.e., pass rates) as the primary measure of curricular 
or instructional change. Assuming no significant variations in students, two reactions are possible to 
course grades. A high failure rate may imply high academic standards, or it may indicate curricular and 
instructional problems. Similarly, an increase in pass rates could be interpreted as the result of lowered 
standards, or it could reflect a more coherent curriculum or improved instructional strategies. The 
problem in using course grades as the primary indicators is that such an approach cannot resolve such 
issues. 

In addition, a primary focus on course pass/fail rates ignores student patterns in terms of withdrawal, 
requesting an incomplete grade, repetition of the course, and other behaviors that indicate whether the 
course is achieving its purpose. See Figure 1 for a flow chart that illustrates the potential outcomes of a 
student taking a gateway course for the first time: 



6 



o 

ERIC 



4 



Paper presented at the 39“* Annual Forum of the Association for Institutional Research (5-31-99) 
* * * Please do not reproduce, disseminate or cite without permission of the author * * * 



Figure 1 



Student Flow through a Gateway Course 
and Potential Outcomes 
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> The Academic Context at UTEP and Faculty Goals for Calculus Curricular Reform 

The University of Texas System Louis Stokes Alliance for Minority Participation (UT-System LSAMP) 
is part of a national initiative funded by the National Science Foundation that aims to increase the number 
of under-represented minority students graduating in Science, Engineering, and Mathematics as part of its 
mandate from the U.S. Congress. One of the major goals of the UT-System LSAMP is to improve the 
introductory courses in those academic programs (sometimes referred to as “gatekeeper” courses) through 
faculty innovation in curricular and instructional practices. 

The anticipated outcome is the increased retention and academic success of all freshmen and sophomore 
students. Changing the role of these courses from a barrier into one of a “gateway” should also contribute 
to an increased number of students who persist in Science and Engineering programs and who graduate in 
a timely fashion. Targeted courses include Precalculus and Calculus, General Biology, General Physics, 
General Chemistry, and Introduction to Engineering. Drawing on a pilot needs assessment project from 
the Puerto Rican LSAMP,"* The University of Texas at El Paso (UTEP) has explored a variety of ways 
that an institutional research office can support academic departments that are experimenting with 
gateway course curricular change and instructional innovation. 

Frustrated by students’ inability to pass Precalculus and Calculus, the chair of the UTEP Department of 
Mathematical Sciences decided to undertake an extensive curricular and instructional reform effort, 
including the implementation of Harvard Calculus Reform beginning in the fall 1994. Simultaneously, a 
full-time lecturer volunteered to design a modular approach to Precalculus.’ The chair realized, however, 
that a longitudinal evaluation model would be required to validate any such initiatives, since colleagues 
might be quick to challenge grade improvements as resulting from a watered-down curriculum and/or 
instructor sympathy. He proposed, therefore, the radical concept that gateway course curricular reform 
could not be evaluated by the course’s pass rates, but rather that the only valid measure would be 
students’ grades in the next course in the curricular sequence - and that if a gateway course were to be 
judged effective, the majority of students who completed it should be able to pass the next course in the 
sequence on the first try. 

Given the math department’s challenge of how to evaluate such a premise, the UTEP Center for Institutional 
Evaluation, Research and Planning began in 1994 to design and pilot the ICE^ (ICE-squared) model to study 
the efficiency and the effectiveness of the Precalculus course. Preliminary results suggest that the model 
provides a more comprehensive and objective means to analyze student academic progress and, based on 
informal feedback from academic leaders, can serve as a valuable catalyst to assist faculty in assessing their 
curricular reform efforts. 

^ Initiating a Dialogue with Faculty about Evaluation of Curricular Reform 

Stark et al. (1997) talk about the need for institutional researchers to work on measures of prorgram 
effectiveness or efficiency and suggest various strategies to use without reducing faculty autonomy. 

The ICE^ process begins with department faculty who have identified a gateway course as a barrier and, 
therefore, have decided to undertake some type of departmentally sponsored innovation. Center dialogue 
with the faculty and/or the chair about the history and conceptual base of the ICE^ model is the first step, 
followed by a demonstration of one course’s outcomes. This leads to the identification of the 
department’s targeted gateway course, the next course(s) in the curricular sequence that faculty expect 
students routinely to take, and the specification of cohorts who will be tracked (i.e., the number of years, 
with three being a recommended period). Agreement also has to be reached about the baseline data, i.e., 
former cohorts or retrospective studies, which will be used for comparison purposes as the reform is 
implemented and about the number of years that cohorts affected by the innovation will be followed. If 
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certain sections of the gateway course are involved in the innovation and others are not, these must be 
clearly identified in advance. See Figure 2 for an illustration of the consultation process. 

It becomes quickly evident that attempting to track the many students enrolled in the large number of 
sections typical of many gateway courses is a formidable task. As faculty buy into the premise of success 
in the next course in the curricular sequence on the first attempt as the most valid measure of their reform 
efforts, they generally recognize that simply tallying grades in the gateway course will be insufficient. 
They may also become very anxious about individual accountability and the possibility of retaliation or 
censure. Clear direction from the chair about access to section information that might affect faculty 
privacy, therefore, is essential. The ICE^ model was not designed to evaluate faculty performance or to 
target individuals for punishment: its goal is to support faculty improvement efforts by providing 
longitudinal and objective measures of curricular reform outcomes. It works best when combined with 
realistic and flexible formative evaluation techniques that provide student feedback to instructors on a 
routine basis. Therefore, as a general guideline, the Center shares all results (i.e., sections and 
summaries) with the department chair so as to facilitate departmental discussion and planning, but only 
summary results with any other interested parties or masked section results for demonstration purposes. 




Figure 2 

The ICE^ Model Dialogue: 

Cooperation between Department Faculty 
and the Institutional Research OfHce 




Department Chair/ 
Faculty identifies: 



Curriculum review begins at 
Departmental Level 
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> UTEP’s PreCalculus Reform Model 

The University’s math department engaged in several years and three phases of curricular innovation in 
an effort to improve student learning in the PreCalculus course as illustrated in Figure 3: 



Figure 3 

UTEP PreCalculus Curricular Reform: 
A Modular Approach 



PreCalculus I 



POot of4SCH 
Modular Approach 
In Some Sections 



Fall 1993 - Summer 1995 



PreCalculus 




Fall 1995 - Summer 1998 



Fall 1998 - Present 



♦ Before Curricular Reform: Students had to complete PreCalculus I and PreCalculus II, each a four 
semester-credit-hour course, before they proceeded to Calculus I. 

♦ Pilot Modular Curriculum for PreCalculus I: This four semester-credit-hour course began a new 
modular approach in fall 1993, with a comprehensive evaluation process that included intensive 
student and instructor feedback. 

• PreCalculus II: For the next five years, students were still required to complete another four 
semester-credit-hour course, PreCalculus II, before they could proceed to Calculus I. 

♦ New Department-wide PreCalculus Course: In fall 1998, the math department created a new 
PreCalculus course with the reform modular curriculum by compressing the former two courses that 
were four semester-credit-hours each into one five semester-credit-hour course. 

♦ Additional Instructional Changes: The university began piloting a cluster course mechanism for 
pre-engineering and pre-science majors in fall 1995. Selected numbers of students were enrolled in 
the same sections of PreCalculus I, English, and an engineering or science course to encourage the 
development of academic peer support groups. 



For more information about the content and process of the reform PreCalculus course, see 

WWW. math. uten.edu/classes/precalculus ^ 
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> Gateway Course Efficiency: Students Get Through 

The ICE index** serves as a measure of a gateway course’s efficiency - that is, it takes into consideration 
the pattern of unsuccessful attempts, incomplete grades, withdrawals, and repetition that often 
characterize many such courses, while tracking students’ ultimate outcome in the targeted course. The 
index demonstrates the progress of cohorts of first-time attempters in the Precalculus course over a two- 
year period by examining: 

• the percent of first-time takers who pass on the initial attempt; 

• the percent of first-time takers who pass eventually; 

• the percent of students unable to complete the course; and 

• the total number and percent of successful outcomes (all students who pass with a 
grade of C or higher, including both first-time takers and repeaters from the initial 
cohort). 

The ideal ICE index for a gateway course would be a low of 1.0, indicating that every student who 
registered for the course passed with a grade of A, B, or C on the first attempt. It thus serves as an 
indirect measure of cost-efficiency for a department chair. As ICE rises, it indicates that the course 
requires increasingly greater institutional resources, as well as student investment of time and funds. 

The index provides a department chair with one objective measure to reflect the true costs (facility 
salaries, faculty frustrations, student attrition, loss of potential majors to programs that require the 
gateway course, etc.) of the gateway course. Calculations of the ICE index illustrate the traps that 
students fall into which may tend to drive them away from the gateway course, thus eliminating them as 
potential majors in fields that require the course. UTEP Precalculus data in Table 1 illustrate this first 
phase of the model, and Figures 4 and 5 graph the outcomes and ICE index: 

Table 1 

UTEP Precalculus Curricular Reform: 

A Two-Year Report on Course Efficiency 
“ Performance in Precalculus by Cohorts of First-Time Students 



Cohorts of First-Time Students: 


Fall 1993 


Fall 1994 


Fall 1995 


Fall 1996 


Total number of first-time takers 


396 


438 


388 


387 


First-time takers who passed Precalculus on 
the first attempt 


14.9% 

(59) 


32.6% 

(143) 


36.7% 

(143) 


49.9% 

(193) 


Additional students in the first-time cohort 
who eventually passed Precalculus 


30.3% 

(120) 


29.9% 

(131) 


28.7% 

(111) 


23.5% 

(91) 


Students who had successful Precalculus 
outcomes (grade of C or above) 


45.2% 

(179) 


62.6% 

(274) 


65.4% 

(254) 


73.4% 

(284) 


ICE: An Indicator of Course Efficiency 


3.83 


2.59 


2.37 


1.98 
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Figure 4 

ICE: Performance in Precalculus by Four Cohorts of 
First-Time Students over the Subsequent Two Years 




□ Cohort student who were unable to pass Precalculus over the two subsequent years 

□ Cohort student who did not pass Precalculus on the first attempt, but eventually passed 

□ Cohort students who passed Precalculus on the first attempt 

Figure 5 



ICE: Index of Course Efficiency 
Performance in Precalculus by First- Time Takers 




> Gateway Course Effectiveness: Students Learned What They Need to Know 

The ICE^ measure serves a different purpose, in that it provides a measure of the gateway course’s 
effectiveness - defined as the student’s success in the next course in the curricular sequence. Therefore, 
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the index follows the original ICE cohorts over a two-year period into the course that the faculty has 
identified as the appropriate next step in their degree plan. ICE^ tracks the Phase I succeeders (i.e., all 
first-time attempters who eventually passed the gateway course) to determine if the course is achieving its 
presumed purpose, i.e., preparing students to pass the next course on their first attempt. The index 
examines: 

• the percent of Phase I successful students who enroll in the next course in the 
curricular sequence - an indicator of whether capable students are continuing in the 
major as a result of the gateway course experience; 

• the percent of first-time takers who pass the next course on the initial attempt: 

an indicator of whether the gateway course prepares students with the knowledge and 
skills needed for the next course in the curricular sequence; and 

• the total number of first-time takers who pass the next course on the initial 
attempt: an indicator of the potential pool of students for majors requiring the 
gateway course and the next course in the curricular sequence. 

An ICE^ index of 1.0 for the targeted gateway course is ideal, indicating that every student who had 
passed the gateway course and subsequently registered for the next course in the curricular sequence 
passed it with a grade of A, B or C on the first attempt, while an index of zero demonstrates that not a 
single student who had been successful in the gateway course was able to pass the next course on the first 
attempt. This is a highly conservative index, reflecting as it does the demanding premise of the 
mathematics chair: one attempt, one success for the majority of students. Thus, a lower ICE^ index 
suggests that the gateway course is not adequately preparing students for the following course. 

For example, the ICE^ model demonstrates the significant strides being made by the UTEP Precalculus 
program, which has been involved in an extensive curricular reform since fall 1994 (see Table 2 and 
Figures 6 and 7). As noted above, the mathematics department identified its target course, decided on a 
particular curricular approach, developed some innovative instructional approaches, and incorporated an 
on-going formative evaluation process whereby students regularly provide feedback to instructors, and 
the coordinator analyzes and compares student progress across sections. The ICE^ model, therefore, is an 
essential component of a major departmental curricular and instructional reform effort.’ 

Although the mathematics sequence actually involved a Precalculus II course, the chair asked that it be 
ignored in the evaluation, since the faculty were considering combining the two Precalculus courses. 
Therefore, the effectiveness of Precalculus I was evaluated with respect to students’ ability to succeed in 
the first Calculus course. An analysis of the success in the UTEP Calculus I course of cohorts of first- 
time students who passed the Precalculus course is effectively summarized by ICE^. 

Curricular review by the Precalculus instructors and by the department chair, therefore, incorporates a 
wide variety of formative evaluation information and the objective indices of progress offered by the ICE^ 
model. For example, based on the ICE index results, the faculty developed such confidence in the new 
approach that they decided to restructure the Precalculus curriculum and to eliminate the Precalculus II 
course, since it was evident that the changes implemented in Precalculus I were significantly improving 
student achievement, particularly in terms of increasing the number of students going on to Calculus I. 
Results for the ICE^ index, however, show that course alignment issues still remain. 
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Table 2 

UTEP Precalculus Curricular Reform: FoIIow-up on Gateway Course Effectiveness 

— Performance in Calculus I over the Subsequent Two Years by 
Cohorts of First-Time Students Who Passed Precalculus. 



Cohorts of First-Time Calculus I Students: 


Fall 1993 


Fall 1994 


Fall 1995 


Fall 1996 


Number of successful Precalculus students 


179 


274 


254 


284 


Successful Precalculus students who enrolled in 
Calculus I 


43.0% 

(77) 


45.5% 

(133) 


46.5% 

(118) 


45.4% 

(129) 


Students who passed Calculus I on the first 
attempt 


58.4% 

(45) 


55.6% 

(74) 


62.2% 

(77) 


56.6% 

(73) 


ICE^: An Indicator of Course Effectiveness - 

- for each cohort of successful Phase 1 Pre- 
calculus students who enrolled in Calculus I 


.58 


.56 


.62 


.57 



Figure 6 



ICE^: Performance in Calculus I over the Subsequent Two Years 
by Cohorts of First- Time Students who Passed Precalculus 




□ Cohort students who passed Calculus I on the first attempt 
H Successful Precalculus students who enrolled in Calculus I 
□Cohort students who passed Precalculus on first or subsequent attempts 
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Figure 7 

ICE^: Index of Gateway Course Effectiveness 
- Performance in Calculus I by Successful Precalculus Students 




Fall 1993 Fall 1994 Fall 1995 Fall 1996 



> Conclusion 

In an NSF workshop on indicators of success in post-secondary SMET education, the director of the 
Puerto Rican LSAMP, Manuel Gomez, proposed that, 

to bring about effective institutionwide use of assessment data, one must approach each 
stakeholder group in terms of its values and needs. . . Only when confronted with data on their 
own students... will faculty buy into the conclusions and start to change their departments... 

He therefore urged reformers to structure assessment and evaluation information intended for 
administrators in ways that clearly communicate how educational change is affecting the 
system. . . Gomez argued that while change makers at the classroom and departmental level are 
essential, isolated individual efforts ultimately will be rejected by the institution if institutional 
leaders do not understand the cumulative value of their efforts. (Millar, 1998, pp. 28-29) 

The two components of the ICE^ model provide valuable information for departmental planning, both to 
department chairs and to faculty. The two indices can serve as longitudinal measures of the benefits of 
curricular change. The information generated from calculation of the ICE index illustrates differences of 
efficiency among sections and can challenge faculty to discuss among themselves why some sections 
generate such high levels of student repetition and withdrawals. If arguments linked to high academic 
standards arise, the ICE^ index encourages more objective discussions of gateway course effectiveness 
and transferable student knowledge and skills. Such dialogue should promote a more carefully aligned 
curriculum and a more objective arena for instructor improvement and student achievement. 

In addition, the ICE" model allows a department chair to compare targeted sections involved in curricular 
or instructional reform with traditional sections, the performance of different groups of students (e.g., the 
subsequent success of students who earn a grade of C in comparison to those with A’s and B’s), and what 
patterns have occurred with respect to student withdrawals or incomplete grades. UTEP’s math 
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department has been engaged in several years of dramatic change in both the Precalculus course and the 
Calculus sequence. This makes evaluation of the impact of curricular reform quite difficult. The ICE^ 
model provides a valuable framework to use in illuminating the effects of different interventions, and 
UTEP is currently engaged in several of these follow-up studies. 

Another potential application of the ICE^ model involves examining the outcomes of a gateway course 
that feeds more than one curricular sequence. For example, the Introductory Biology course typically 
enrolls students who are interested in becoming Biology majors and others focused on the health 
professions, as well as others who may simply be looking for a science course to complete core 
curriculum requirements. UTEP is in the process of adapting the ICE^ model to examine these different 
curricular tracks so that faculty have a more comprehensive measure to evaluate the effectiveness of their 
curriculum. 

Institutions that are concerned improving freshmen retention rates, increasing the number of students or 
of particular groups of students who enter fields with challenging gateway courses, raising their 
graduation rates, and/or demonstrating that their graduates’ have attained specific knowledge and skills 
should find the ICE^ model to be useful. Additional information about the ICE^ model and about 
calculations of the two indices is available on the web site of the AMP Virtual Center on Formative 
Evaluation, which is funded by the National Science Foundation and managed by the UTEP Center for 
Institutional Evaluation, Research and Planning: httD://ampvi.utep.edu ^ 

Partial funding for the development of the evaluation model described here came from the National Science 
Foundation’s Louis Stokes Alliance for Minority Participation (AMP) program (NSF grant #HRD 92551660). 



’ The National Science Foundation defines under-represented minority students as individuals of African-American, 
Hispanic, or Native American origin. 

“* Ana C. Pinero, “Measuring the Effectiveness and Efficiency of the SMET Undergraduate Enterprise Through a 
Retrospective Cohort Study,” The Puerto Rico Louis Stokes Alliance for Minority Participation - A Systemic Effort 
Sponsored by NSF, July 1996. 

^ Dr. Simon Bernau, currently Dean of the School of Science at California State Polytechnic University-Pomona, 
was chair of the UTEP Department of Mathematical Sciences at the time. Dr. Nancy Marcus designed and 
implemented the UTEP Precalculus modular program and continues to coordinate it. The author expresses her 
appreciation to them both for their vision of UTEP student success and for their support in developing this 
evaluation model. 

'■ The UTEP Precalculus curricular project is coordinated by Dr. Nancy Marcus. For more information on the 
curricular and instructional approach, visit the web site at www.math.uteD.edu/classes/precalculus . 

’ The development of the web site of the Virtual Center for Formative Evaluation was funded by the National 
Science Foundation, sponsored by The University of Texas System Louis Stokes Alliance for Minority 
Participation, and administered by the UTEP Center for Institutional Evaluation, Research and Planning. 
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